Skip to content

Add: overlap HBG successor preparation with active execution - #1587

Merged
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
Crane-Liu:codex/worker-async-b4b-hbg-prepared-epoch
Aug 3, 2026
Merged

Add: overlap HBG successor preparation with active execution#1587
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
Crane-Liu:codex/worker-async-b4b-hbg-prepared-epoch

Conversation

@Crane-Liu

@Crane-Liu Crane-Liu commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Build W2 on top of Add: bounded asynchronous native run lane #1650 instead of carrying the common native lane inside this PR.
  • Opt only host-build-graph into concurrent successor preparation by using the inactive HOST_PER_RUN arena bank.
  • Keep launch ownership FIFO and force diagnostic/incompatible runs back to depth one.
  • Add delayed-kernel real-device coverage proving successor prepare overlaps predecessor device execution while launch stays sequential.

Dependency

This PR is intentionally stacked on #1650. Merge #1650 first; after that this PR can be rebased to a single W2 commit on main.

Validation

  • Pre-commit: all hooks passed for the eight changed files.
  • Real hardware, A2A3, device 7: task_20260803_005853_170545415324
    • worker_async_fifo W2 scenarios: 3 passed
    • the combined lifecycle case exposed a test-only generation collision after manual native API use; corrected by keeping those manual epochs generation-stable.
  • Real hardware, A2A3, device 7: task_20260803_010335_231693332344
    • native_run_lifecycle: 1 passed
  • Tested tree hash against upstream/main: de9d55a628f988a047084615dda6c18f6c3a7f3086875b0519894ff587f388d0.

No simulator validation was run; hardware validation is the acceptance evidence for this stage.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d72f9778-6112-4c90-a6e1-125a1d19aee5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The change adds generation-safe pipeline-slot admission, run-scoped FIFO scheduling, and optional two-frame prepared activation across hierarchical workers. Mailbox layouts, runtime APIs, Python bindings, endpoint dispatch, launch-shape activation, and concurrency tests are updated accordingly.

Changes

Whole-run admission and scheduling

Layer / File(s) Summary
Run contracts and admission
src/common/hierarchical/types.*, src/common/hierarchical/orchestrator.*, src/common/worker/pipeline_slot_pool.h
Runs acquire generation-safe pipeline leases, track lifecycle phases and task slots, and release leases during terminal retirement.
Run-scoped dispatch
src/common/hierarchical/scheduler.*, src/common/hierarchical/worker.*
Ready queues and scheduler dispatch are partitioned by active/preparable run IDs, with FIFO-head activation and capacity-aware dispatch.
Prepared endpoint execution
src/common/hierarchical/worker_manager.*, src/common/worker/chip_worker.*
Endpoints support optional prepared execution with two mailbox frames, backend-ready fencing, explicit activation, identity validation, and abort cleanup.

Runtime and Python integration

Layer / File(s) Summary
Mailbox and runtime ABI
src/common/worker/pto_runtime_c_api.h, src/common/platform/onboard/host/c_api_shared.cpp, src/common/platform/onboard/host/device_runner_base.*
The runtime exposes prepared-run capability and lifecycle functions, validates PreparedRunIdentity, and activates launch geometry immediately before execution.
Worker startup and bindings
python/simpler/worker.py, python/bindings/*, python/simpler/task_interface.py
Startup negotiates per-chip depth and prepared-activation support, configures endpoints and frame counts, propagates leases, and releases the GIL around native execution.
Validation and documentation
tests/ut/*, tests/st/*, docs/task-flow.md
Tests cover FIFO admission, queue ordering, prepared activation, mailbox identity, capability negotiation, and end-to-end two-frame execution; task-flow documentation describes the updated protocol.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Poem

A rabbit saw leases hop in line,
Through mailbox frames in neat design.
Prepare, then activate bright,
FIFO bounds each run’s flight.
“Nibble-safe dispatch!” I cheer—
The next task waits, then races clear.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.64% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: overlapping HBG successor preparation with active execution.
Description check ✅ Passed The description directly explains the prepared-run overlap, FIFO execution, identity handling, compatibility behavior, and validation results.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 9

🧹 Nitpick comments (10)
src/common/hierarchical/worker_manager.cpp (1)

305-315: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

enqueue_dispatch transiently overshoots capacity.

fetch_add then rollback means a concurrent has_capacity() / idle() observer can see inflight_ == capacity_ + 1 before the rollback lands, so a scheduler gate keyed on has_capacity() may reject a slot that is actually free (and, symmetrically, idle() never reports true spuriously, so the risk is only a missed admission). A CAS loop keeps the counter within bounds.

♻️ Bounded reservation
-    uint32_t previous = inflight_.fetch_add(1, std::memory_order_acq_rel);
-    if (previous >= capacity_) {
-        inflight_.fetch_sub(1, std::memory_order_acq_rel);
-        throw std::logic_error("WorkerThread::dispatch: endpoint capacity exceeded");
-    }
+    uint32_t previous = inflight_.load(std::memory_order_acquire);
+    do {
+        if (previous >= capacity_) {
+            throw std::logic_error("WorkerThread::dispatch: endpoint capacity exceeded");
+        }
+    } while (!inflight_.compare_exchange_weak(
+        previous, previous + 1, std::memory_order_acq_rel, std::memory_order_acquire
+    ));
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/hierarchical/worker_manager.cpp` around lines 305 - 315, Update
WorkerThread::enqueue_dispatch to reserve inflight_ with a compare-exchange loop
that only increments when the current value is below capacity_. Remove the
fetch_add-and-rollback sequence so concurrent has_capacity() and idle()
observers never see inflight_ exceed capacity_; retain the existing exception,
dispatch ID assignment, queue insertion, and notification behavior.
src/common/hierarchical/worker_manager.h (1)

97-97: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Declare the protocol version with its wire width.

The frame trailer field is 8 bytes (worker_manager.cpp writes a uint64_t; Python unpacks =Q), so a uint32_t constant forces an implicit widen at every use and invites a 4-byte memcpy if someone passes the constant directly.

♻️ Match the wire type
-static constexpr uint32_t MAILBOX_TASK_PROTOCOL_VERSION = 2;
+static constexpr uint64_t MAILBOX_TASK_PROTOCOL_VERSION = 2;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/hierarchical/worker_manager.h` at line 97, Change
MAILBOX_TASK_PROTOCOL_VERSION from uint32_t to uint64_t so its declared type
matches the 8-byte protocol trailer written by the worker-manager serialization
path and read as =Q by Python. Keep its value and existing uses unchanged.
python/simpler/worker.py (4)

1753-1761: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

slot_id < task_frame_count is validated against the parameter, but the frame arrays are sized by _TASK_FRAME_COUNT.

frame_bufs / frame_addrs are built with range(_TASK_FRAME_COUNT) (Line 1740-1742) while identity validation bounds slot_id by task_frame_count. A task_frame_count > _TASK_FRAME_COUNT would admit a slot id that indexes past both lists. Unreachable today (the only caller passes _TASK_FRAME_COUNT), but the two sources of truth should be one.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/simpler/worker.py` around lines 1753 - 1761, The validate_identity
function must bound slot_id using the same _TASK_FRAME_COUNT constant used to
size frame_bufs and frame_addrs, rather than the task_frame_count parameter.
Update that validation while preserving the other identity checks.

1888-1892: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Silently swallowed abort_prepared failures leave no diagnostic trail.

All four abort paths use bare except Exception: pass. An abort that fails means the backend still holds an unpublished prepared run (device arena/slot not reclaimed), which will surface later as an unexplained slot exhaustion or lease mismatch with no clue about the original cause. Emit the exception to stderr like the other best-effort cleanups in this module do.

♻️ Suggested logging
-                        try:
-                            abort_prepared(prepared_identity)
-                        except Exception:  # noqa: BLE001
-                            pass
+                        try:
+                            abort_prepared(prepared_identity)
+                        except Exception as exc:  # noqa: BLE001
+                            sys.stderr.write(
+                                f"chip_process dev={device_id}: abort_prepared failed: {type(exc).__name__}: {exc}\n"
+                            )
+                            sys.stderr.flush()

Also applies to: 1908-1911, 1949-1953, 1973-1977

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/simpler/worker.py` around lines 1888 - 1892, Update all four abort
cleanup paths around abort_prepared to preserve best-effort exception handling
while emitting each caught exception to stderr, matching the diagnostic pattern
used by other best-effort cleanups in the module. Replace the silent pass blocks
associated with prepared_identity in each path; do not alter the abort flow or
re-raise the failures.

Source: Linters/SAST tools


1737-1737: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

prepared_frames is mutated from both the admission and executor threads without synchronization.

The admission thread inserts/pops at Lines 1881/1893/1907 while the executor pops at Lines 1954/1960. Individual dict ops are atomic in CPython, so nothing corrupts today, but the check-then-act pairs (index in prepared_frames → insert, get → compare → pop) are not, and the invariant "one prepared epoch per slot" depends on mailbox-state ordering rather than on any explicit guard. Moving these accesses under action_cv (already held for the queue) makes the ownership rule enforceable rather than incidental.

Also applies to: 1906-1921, 1945-1960

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/simpler/worker.py` at line 1737, Protect all accesses to
prepared_frames in the admission and executor paths with action_cv, including
the check-then-insert/pop sequences and get/compare/pop logic around the
identified admission and executor operations. Ensure each compound operation
executes while holding the condition’s lock, preserving the
one-prepared-epoch-per-slot invariant without changing mailbox ordering.

1836-1854: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Admission thread spins with no yield.

admission_loop polls the control word and both frame states in a tight loop with no time.sleep/backoff. In the forked chip child this burns a core per chip and contends for the GIL with the executor thread between its native calls (the executor only drops the GIL inside run_from_blob / execute_prepared). A short bounded sleep once no frame is actionable would keep latency while removing the busy-wait.

♻️ Suggested bounded poll
         def admission_loop() -> None:
             nonlocal control_queued
             while not stop_admission.is_set():
+                progressed = False
                 control_state = _mailbox_load_i32(state_addr)

…and at the end of the frame sweep, if not progressed: time.sleep(_MAILBOX_POLL_INTERVAL_S).

Please confirm what poll cadence the existing _run_mailbox_loop uses so the two loops stay consistent.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/simpler/worker.py` around lines 1836 - 1854, Update admission_loop to
avoid tight polling by tracking whether control or frame admission made progress
during each iteration and sleeping for the established mailbox poll interval
when none did. Reuse the cadence used by _run_mailbox_loop via
_MAILBOX_POLL_INTERVAL_S, while preserving immediate handling of actionable
frames and shutdown/control requests.
tests/ut/py/test_callable_identity.py (1)

604-604: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Assert against the constant rather than the literal 2.

This worker has no device_ids, so _start_hierarchical leaves direct_chip_pipeline_depth at PTO_PIPELINE_MAX_DEPTH. Hard-coding 2 makes the test fail confusingly if that cap ever changes, and hides what the assertion is actually about (no chips ⇒ no depth negotiation, so the cap is passed through).

♻️ Proposed change
-        assert fake_c_worker.pipeline_depth == 2
+        # No device_ids, so no chip negotiation happens and the cap is passed through.
+        assert fake_c_worker.pipeline_depth == worker_mod.PTO_PIPELINE_MAX_DEPTH
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/py/test_callable_identity.py` at line 604, Update the assertion for
fake_c_worker.pipeline_depth to compare against PTO_PIPELINE_MAX_DEPTH instead
of the literal 2, preserving the test’s verification that the no-device path
passes through the configured maximum depth.
tests/ut/cpp/hierarchical/test_scheduler.cpp (1)

721-722: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Prefer a bounded copy into the fixed-size output_prefix.

std::strcpy is flagged by static analysis; std::snprintf(diagnostic_config.output_prefix, sizeof(diagnostic_config.output_prefix), "%s", "/tmp/simpler-diagnostic-successor") keeps the intent and removes the unbounded-write pattern.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/cpp/hierarchical/test_scheduler.cpp` around lines 721 - 722, Replace
the unbounded std::strcpy assignment to diagnostic_config.output_prefix with a
bounded std::snprintf call using sizeof(diagnostic_config.output_prefix),
preserving the existing prefix value and fixed-buffer safety.

Source: Linters/SAST tools

tests/st/a2a3/host_build_graph/worker_async_fifo/kernels/orchestration/pipelined_vector_orch.cpp (1)

18-20: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Duplicated orchestration kernel differing only in kChainLength.

This file is otherwise identical to tests/st/a2a3/host_build_graph/worker_async_endpoint/kernels/orchestration/long_vector_orch.cpp; consider a single shared source with the chain length injected as a compile definition.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@tests/st/a2a3/host_build_graph/worker_async_fifo/kernels/orchestration/pipelined_vector_orch.cpp`
around lines 18 - 20, Consolidate the duplicated orchestration kernel by reusing
the shared implementation from long_vector_orch.cpp, and remove the duplicate
source-specific logic from the worker_async_fifo path. Inject the differing
kChainLength value of 512 through the build configuration as a compile
definition, while preserving the existing kernel behavior and constants.
tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py (1)

161-169: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Capture the submitter thread's exception for a diagnosable failure.

If st_worker.submit(third_graph) raises inside the daemon thread, the exception is discarded and the test fails at Line 169 with a misleading "did not enter after the first run freed its slot" message.

♻️ Record the failure
-            submitter = threading.Thread(
-                target=lambda: third_result.setdefault("handle", st_worker.submit(third_graph)), daemon=True
-            )
+            def _submit_third():
+                try:
+                    third_result["handle"] = st_worker.submit(third_graph)
+                except BaseException as exc:  # noqa: BLE001
+                    third_result["error"] = exc
+
+            submitter = threading.Thread(target=_submit_third, daemon=True)

and assert third_result.get("error") is None before Line 176.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py`
around lines 161 - 169, Update the third-graph submitter thread around
st_worker.submit to catch any exception and store it in third_result["error"]
alongside the handle. Before the existing post-release callback assertion,
assert that third_result.get("error") is None so submission failures are
reported directly while preserving the current admission-capacity checks.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/common/hierarchical/orchestrator.cpp`:
- Around line 355-369: Update Orchestrator::cancel_unstarted_run to avoid
calling try_consume more than once for a FAILED slot whose normal completion
path already consumed it. Gate the failed-slot pass and producer pass using the
existing consumption state, or otherwise track fanout-reference release
explicitly, while preserving cancellation cleanup for unconsumed slots and
preventing the total + 1 threshold from being reached prematurely.
- Around line 345-354: Update the cancellation loop in cancel_unstarted_run so
failure_message is written only after successfully transitioning the slot from
PENDING or READY to FAILED via the existing CAS; do not assign it before the
CAS. Ensure the completion/failure reporting path synchronizes access to
failure_message as needed, using the existing fanout_mu consistently if required
by the current state ownership.

In `@src/common/hierarchical/scheduler.cpp`:
- Around line 185-189: Align the preparable wake predicate in scheduler.cpp
lines 185-189 with dispatcher acceptance by checking supports_prepare_activate,
has_capacity(), and diagnostics_any(), or use a per-run declined latch. In the
group dispatcher at lines 380-386, pop and discard stale non-READY heads before
continuing. In the single-run dispatcher at lines 416-427, discard non-READY
entries and continue; reserve enqueue_ready_cb for genuine run or routing
mismatches.

In `@src/common/hierarchical/worker_manager.cpp`:
- Around line 413-452: Ensure dispatch IDs assigned by enqueue_dispatch cannot
remain unretired when WorkerThread::dispatch_process fails before
endpoint_->run_prepared_with_activation. Add or invoke an endpoint-side
abandonment path for the assigned ID on the null-endpoint, invalid-slot, and
other pre-endpoint failure paths, advancing the same publish sequence used by
retire_publish_sequence; alternatively move ID assignment into the endpoint so
only entered dispatches receive IDs. Preserve normal run_two_frame publication
behavior.

In `@src/common/platform/onboard/host/c_api_shared.cpp`:
- Around line 675-682: Update format_prepared_attrs to remove the request and
epoch trace attributes, since PreparedRunIdentity has no corresponding fields;
emit only run, slot, generation, and dispatch using their actual identity
members.

In `@src/common/worker/chip_worker.cpp`:
- Around line 650-670: Update ChipWorker::abort_prepared to claim the validated
slot under prepared_slots_mu_ before releasing the lock, transitioning it from
PREPARED to the same in-progress state used by execute_prepared. Recheck the
lease identity while claiming, then perform select_slot_resources and
abort_prepared_fn_ only after ownership is acquired, preventing concurrent abort
callers from processing the same slot.

In
`@tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py`:
- Around line 113-122: Make the frame-state assertion deterministic by blocking
frame A at the appropriate synchronization point, using the release-fence
pattern from test_worker_async_fifo, until frame B reaches _TASK_ACCEPTED_STATE
while A remains _TASK_ACTIVE. Update the polling loop around
saw_active_and_accepted to include a short sleep to avoid busy-spinning and GIL
contention, then release the block before run.wait while preserving the existing
assertion.

In `@tests/ut/cpp/hierarchical/test_orchestrator.cpp`:
- Around line 761-772: Replace the fatal readiness assertion in
tests/ut/cpp/hierarchical/test_orchestrator.cpp:761-772 with a non-fatal check
and unconditionally release the prepared lease via the existing recovery path
before continuing or returning. Apply the same change at
tests/ut/cpp/hierarchical/test_orchestrator.cpp:808-817, ensuring the active
slot is always consumed so replacement can complete. At
tests/ut/cpp/hierarchical/test_scheduler.cpp:541-548, replace ASSERT_NE with
EXPECT_NE and skip recovery dispatch when the prerequisite is unavailable rather
than returning while the future remains outstanding.

In `@tests/ut/cpp/hierarchical/test_scheduler.cpp`:
- Around line 578-590: Bound both mailbox polling loops in the mock child thread
around the child lambda with steady-clock deadlines, and exit the child when
PREPARE_READY or ACTIVATE is not observed before its deadline. Ensure
child.join() can always complete while preserving the existing state transitions
when each expected state arrives.

---

Nitpick comments:
In `@python/simpler/worker.py`:
- Around line 1753-1761: The validate_identity function must bound slot_id using
the same _TASK_FRAME_COUNT constant used to size frame_bufs and frame_addrs,
rather than the task_frame_count parameter. Update that validation while
preserving the other identity checks.
- Around line 1888-1892: Update all four abort cleanup paths around
abort_prepared to preserve best-effort exception handling while emitting each
caught exception to stderr, matching the diagnostic pattern used by other
best-effort cleanups in the module. Replace the silent pass blocks associated
with prepared_identity in each path; do not alter the abort flow or re-raise the
failures.
- Line 1737: Protect all accesses to prepared_frames in the admission and
executor paths with action_cv, including the check-then-insert/pop sequences and
get/compare/pop logic around the identified admission and executor operations.
Ensure each compound operation executes while holding the condition’s lock,
preserving the one-prepared-epoch-per-slot invariant without changing mailbox
ordering.
- Around line 1836-1854: Update admission_loop to avoid tight polling by
tracking whether control or frame admission made progress during each iteration
and sleeping for the established mailbox poll interval when none did. Reuse the
cadence used by _run_mailbox_loop via _MAILBOX_POLL_INTERVAL_S, while preserving
immediate handling of actionable frames and shutdown/control requests.

In `@src/common/hierarchical/worker_manager.cpp`:
- Around line 305-315: Update WorkerThread::enqueue_dispatch to reserve
inflight_ with a compare-exchange loop that only increments when the current
value is below capacity_. Remove the fetch_add-and-rollback sequence so
concurrent has_capacity() and idle() observers never see inflight_ exceed
capacity_; retain the existing exception, dispatch ID assignment, queue
insertion, and notification behavior.

In `@src/common/hierarchical/worker_manager.h`:
- Line 97: Change MAILBOX_TASK_PROTOCOL_VERSION from uint32_t to uint64_t so its
declared type matches the 8-byte protocol trailer written by the worker-manager
serialization path and read as =Q by Python. Keep its value and existing uses
unchanged.

In
`@tests/st/a2a3/host_build_graph/worker_async_fifo/kernels/orchestration/pipelined_vector_orch.cpp`:
- Around line 18-20: Consolidate the duplicated orchestration kernel by reusing
the shared implementation from long_vector_orch.cpp, and remove the duplicate
source-specific logic from the worker_async_fifo path. Inject the differing
kChainLength value of 512 through the build configuration as a compile
definition, while preserving the existing kernel behavior and constants.

In `@tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py`:
- Around line 161-169: Update the third-graph submitter thread around
st_worker.submit to catch any exception and store it in third_result["error"]
alongside the handle. Before the existing post-release callback assertion,
assert that third_result.get("error") is None so submission failures are
reported directly while preserving the current admission-capacity checks.

In `@tests/ut/cpp/hierarchical/test_scheduler.cpp`:
- Around line 721-722: Replace the unbounded std::strcpy assignment to
diagnostic_config.output_prefix with a bounded std::snprintf call using
sizeof(diagnostic_config.output_prefix), preserving the existing prefix value
and fixed-buffer safety.

In `@tests/ut/py/test_callable_identity.py`:
- Line 604: Update the assertion for fake_c_worker.pipeline_depth to compare
against PTO_PIPELINE_MAX_DEPTH instead of the literal 2, preserving the test’s
verification that the no-device path passes through the configured maximum
depth.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b2216452-97fc-49eb-a3db-87a32be75912

📥 Commits

Reviewing files that changed from the base of the PR and between 33e5450 and fb31c05.

📒 Files selected for processing (33)
  • docs/task-flow.md
  • python/bindings/task_interface.cpp
  • python/bindings/worker_bind.h
  • python/simpler/task_interface.py
  • python/simpler/worker.py
  • src/a2a3/platform/onboard/host/device_runner.h
  • src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp
  • src/common/hierarchical/orchestrator.cpp
  • src/common/hierarchical/orchestrator.h
  • src/common/hierarchical/scheduler.cpp
  • src/common/hierarchical/scheduler.h
  • src/common/hierarchical/types.cpp
  • src/common/hierarchical/types.h
  • src/common/hierarchical/worker.cpp
  • src/common/hierarchical/worker.h
  • src/common/hierarchical/worker_manager.cpp
  • src/common/hierarchical/worker_manager.h
  • src/common/platform/onboard/host/c_api_shared.cpp
  • src/common/platform/onboard/host/device_runner_base.cpp
  • src/common/platform/onboard/host/device_runner_base.h
  • src/common/worker/chip_worker.cpp
  • src/common/worker/chip_worker.h
  • src/common/worker/pipeline_slot_pool.h
  • src/common/worker/pto_runtime_c_api.h
  • tests/st/a2a3/host_build_graph/worker_async_endpoint/kernels/orchestration/long_vector_orch.cpp
  • tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py
  • tests/st/a2a3/host_build_graph/worker_async_fifo/kernels/orchestration/pipelined_vector_orch.cpp
  • tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py
  • tests/ut/cpp/hierarchical/test_orchestrator.cpp
  • tests/ut/cpp/hierarchical/test_pipeline_contract.cpp
  • tests/ut/cpp/hierarchical/test_scheduler.cpp
  • tests/ut/py/test_callable_identity.py
  • tests/ut/py/test_worker/test_host_worker.py

Comment thread src/common/hierarchical/orchestrator.cpp Outdated
Comment thread src/common/hierarchical/orchestrator.cpp Outdated
Comment thread src/common/hierarchical/scheduler.cpp Outdated
Comment thread src/common/hierarchical/worker_manager.cpp Outdated
Comment thread src/common/platform/onboard/host/c_api_shared.cpp Outdated
Comment thread src/common/worker/chip_worker.cpp Outdated
Comment thread tests/ut/cpp/hierarchical/test_orchestrator.cpp Outdated
Comment thread tests/ut/cpp/hierarchical/test_scheduler.cpp Outdated
@ChaoWao
ChaoWao force-pushed the codex/worker-async-b4b-hbg-prepared-epoch branch from fb31c05 to ca8a324 Compare August 3, 2026 04:41
@ChaoWao ChaoWao changed the title Add HBG prepared epoch pipeline Add: overlap HBG successor preparation with active execution Aug 3, 2026
@ChaoWao
ChaoWao force-pushed the codex/worker-async-b4b-hbg-prepared-epoch branch from ca8a324 to 2ee4e9d Compare August 3, 2026 05:06
- Prepare one HBG successor in a distinct lease-selected slot and arena bank while its predecessor executes.
- Keep device launch FIFO-serial and use validation-only staging for diagnostics, TMR, simulation, and unsupported backends.
- Carry generation-bound identity through native prepare, launch, tracing, failure, finalization, and teardown.
- Require a uniform host-runtime pipeline ABI, with explicit depth-one A5 contracts and sim capability adapters, so mismatched DSOs fail during init.
- Resolve expected scheduler admission failures without throwing from the scheduler thread.
- Keep per-thread run selection safe when the host runtime DSO unloads.
- Cover overlap plus the eight-runtime symbol matrix, cancellation, diagnostics, cleanup, and protocol invariants.
@ChaoWao
ChaoWao force-pushed the codex/worker-async-b4b-hbg-prepared-epoch branch from 2ee4e9d to b09f24b Compare August 3, 2026 07:36
@ChaoWao

ChaoWao commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

/run-cpu

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

/run-cpu lane finished with failurehttps://github.com/hw-native-sys/simpler/actions/runs/30795021080

  • ✅ detect-changes
  • ❌ pre-commit
  • ✅ ut-a2a3
  • ✅ ut
  • ✅ packaging
  • ✅ st-sim-a5
  • ✅ st-sim-a2a3
  • ✅ st-onboard-a5
  • ✅ profiling-flags-smoke
  • ✅ st-onboard-a2a3
  • ✅ ut-a5

@ChaoWao
ChaoWao merged commit 2a650f2 into hw-native-sys:main Aug 3, 2026
18 checks passed
ChaoWao added a commit to ChaoWao/simpler-fork that referenced this pull request Aug 3, 2026
A ChipTask's sticky acceptance word is written by the platform runner once
the run crosses its launch boundary, through the set_task_accepted_state_ctx
binding ChipWorker resolves at init. Nothing asserted that it arrives: the
two endpoint scene tests that read the word assert it is still 0 before
activation, and both are a2a3-onboard-only, so the sim side of the binding
had no coverage at all.

That gap hid a real defect until hw-native-sys#1587. The sim c_api exported no
set_task_accepted_state_ctx, so ChipWorker's then-optional load produced
nullptr, both bind sites were skipped, and SimDeviceRunnerBase's
publish_task_accepted stored through a null pointer target. A sim child
therefore never published acceptance, and the run-level fence
(decrement_run_accepts, reached via LocalMailboxEndpoint::read_task_accepted)
advanced only when the run reached a terminal phase — the launch fence
silently degraded into a completion fence.

The test dispatches one ChipTask and asserts the word is set in whichever
mailbox frame carried it, which holds on both endpoint shapes: the parent
clears the word only when it publishes the next task into that frame.
Verified to fail against the pre-hw-native-sys#1587 sim c_api with
"the chip worker never published launch acceptance: [0, 0, 0]".
Crane-Liu added a commit to Crane-Liu/simpler that referenced this pull request Aug 3, 2026
Correct the combined implementation merged by hw-native-sys#1587 to the reviewed v2 architecture while preserving the HBG inactive-bank capability. Keep pipeline metadata optional for older runtimes, centralize generation-bound public handles in Worker, and remove simulator-side capability duplication. The resulting tree exactly matches the real-device validated W1+W2 state and does not introduce RequestSession.
ChaoWao added a commit that referenced this pull request Aug 3, 2026
A ChipTask's sticky acceptance word is written by the platform runner once
the run crosses its launch boundary, through the set_task_accepted_state_ctx
binding ChipWorker resolves at init. Nothing asserted that it arrives: the
two endpoint scene tests that read the word assert it is still 0 before
activation, and both are a2a3-onboard-only, so the sim side of the binding
had no coverage at all.

That gap hid a real defect until #1587. The sim c_api exported no
set_task_accepted_state_ctx, so ChipWorker's then-optional load produced
nullptr, both bind sites were skipped, and SimDeviceRunnerBase's
publish_task_accepted stored through a null pointer target. A sim child
therefore never published acceptance, and the run-level fence
(decrement_run_accepts, reached via LocalMailboxEndpoint::read_task_accepted)
advanced only when the run reached a terminal phase — the launch fence
silently degraded into a completion fence.

The test dispatches one ChipTask and asserts the word is set in whichever
mailbox frame carried it, which holds on both endpoint shapes: the parent
clears the word only when it publishes the next task into that frame.
Verified to fail against the pre-#1587 sim c_api with
"the chip worker never published launch acceptance: [0, 0, 0]".
ChaoWao added a commit to ChaoWao/simpler-fork that referenced this pull request Aug 3, 2026
hw-native-sys#1587 moved three contracts without moving the text that described them, and
left one failure mode expressed as a noexcept violation.

Geometry. resolve_block_dim() and prepare_launch_shape() no longer write
block_dim_ or worker_count_; activate_launch_shape() latches both on the
executor thread immediately before run(). The comment and the LOG_ERROR in
each onboard run() still named prepare_launch_shape, so the one diagnostic a
future reader greps pointed at a function that latches nothing. hw-native-sys#1521 later
edited the line directly below that comment and left it standing, which is
how a stale comment survives. The simulation runners keep their wording:
SimDeviceRunnerBase::prepare_launch_shape does still assign block_dim_.

Streams. RunStreamSlots became a two-thread class when native prepare started
provisioning the successor's slot while the executor retires the
predecessor's. Per-slot handles are safe — admission gives each slot one
owner — but created_count_ is shared across owners and is also read from an
unrelated thread through get_run_stream_set_create_count, so it is now
atomic and the ownership rule is stated on the class.

Thread selection. restore_native_run_thread_selection was noexcept while
run_selection() could throw: on a thread created by create_thread the
per-thread block does not exist yet, so installation allocates. Split out a
non-throwing try_run_selection() and let restore abort with a message on the
unrecoverable path. Returning instead would leave the thread on the default
slot and bank, addressing storage another lease owns, and a freshly started
thread has no channel to report the failure through. B6c removes the
mechanism outright; until then the failure is diagnosable rather than a bare
terminate.

Symbol loading. Since every required pipeline symbol became a strict load,
the dominant cause of a dlsym failure is a host runtime out of sync with the
tree that consumes it. Say so in the error, which otherwise reports only the
missing name.

Also spell the successor-already-staged test as occupied > 1, since the loop
above it has already rejected every predecessor that may not carry one, and
record that simulation discards native-run identity by design.
ChaoWao added a commit that referenced this pull request Aug 3, 2026
)

#1587 moved three contracts without moving the text that described them, and
left one failure mode expressed as a noexcept violation.

Geometry. resolve_block_dim() and prepare_launch_shape() no longer write
block_dim_ or worker_count_; activate_launch_shape() latches both on the
executor thread immediately before run(). The comment and the LOG_ERROR in
each onboard run() still named prepare_launch_shape, so the one diagnostic a
future reader greps pointed at a function that latches nothing. #1521 later
edited the line directly below that comment and left it standing, which is
how a stale comment survives. The simulation runners keep their wording:
SimDeviceRunnerBase::prepare_launch_shape does still assign block_dim_.

Streams. RunStreamSlots became a two-thread class when native prepare started
provisioning the successor's slot while the executor retires the
predecessor's. Per-slot handles are safe — admission gives each slot one
owner — but created_count_ is shared across owners and is also read from an
unrelated thread through get_run_stream_set_create_count, so it is now
atomic and the ownership rule is stated on the class.

Thread selection. restore_native_run_thread_selection was noexcept while
run_selection() could throw: on a thread created by create_thread the
per-thread block does not exist yet, so installation allocates. Split out a
non-throwing try_run_selection() and let restore abort with a message on the
unrecoverable path. Returning instead would leave the thread on the default
slot and bank, addressing storage another lease owns, and a freshly started
thread has no channel to report the failure through. B6c removes the
mechanism outright; until then the failure is diagnosable rather than a bare
terminate.

Symbol loading. Since every required pipeline symbol became a strict load,
the dominant cause of a dlsym failure is a host runtime out of sync with the
tree that consumes it. Say so in the error, which otherwise reports only the
missing name.

Also spell the successor-already-staged test as occupied > 1, since the loop
above it has already rejected every predecessor that may not carry one, and
record that simulation discards native-run identity by design.
Crane-Liu added a commit to Crane-Liu/simpler that referenced this pull request Aug 3, 2026
Rebase the common active-plus-prepared ownership on current main without weakening the uniform pipeline ABI from hw-native-sys#1587 or the runner geometry, stream, and TLS contracts from hw-native-sys#1653. Add generation-bound direct L2 RunHandles, bounded two-slot admission, launch-only acceptance waiting, and deterministic depth-one fallback while preserving HBG inactive-bank preparation. Remove timing-dependent endpoint assertions and keep RequestSession absent.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants